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Motion estimation 



The invention relates to a method and device for estimating motion vectors, and 
to a video display apparatus comprising such a device. 

EP-B^,410,826 [1] discloses a method of performing a recursive motion 
5 estimation between a reference image and a current image, to realize a fixed number of 
iterations, each iteration consisting of: 

- sampling both images with a resolution which increases at each iteration; 

- detecting characteristic samples in the reference image; 

- estimating a local motion vector for each characteristic sample in the reference image, in 
10 which the estimation is initialized witii a displacement vector derived from a global motion 

estimation realized in tiie course of the iteration preceding the current iteration; and 

- carrying out a global motion estimation of the current image, in which values of parameters 
of a model are calculated on Uie basis of the local motion vectors estimation in the course of 
the latest iteration. 

15 To estimate tiie global motion, Uie values of 8 parameters al .. a8 defining a 

motion model are calculated, in such a manner tiiat a sample having coordinates X, Y if 

displaced to coordinates X', Y', where 

X' = ( al.X + a2.Y + a3 ) / ( a7.X + aS.Y + 1 ) and 

Y' = ( a4.X + aS.Y + a6 ) / ( a7.X + aS.Y + 1 ), 
20 Uie 8 parameters al .. a8 being calculated by solving a system of equations with 8 unknown 
which is constituted by the two relationships above, written for characteristic samples, tii 
coordinates X', Y' being determined as a function of an estimated local motion vector fc 
each of these samples and as a function of the coordinates X, Y of the sample undi 
consideration. 

25 To estimate a local motion vector for each characteristic sample of tiie curre: 

image, tiie following steps are carried out: 

- defining a first block of samples of the reference image, centered on tiie characterisr 
smiple under consideration; 

- defining a pluraUty of second blocks of samples of tiic current image, centered on sampl 
30 corresponding to tiie characteristic sample under consideration but shifted over a plurality 
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vectors to be tested, each vector to be tested being constituted by the vector sum of an initial 
displacement of the characteristic sample under consideration calculated from the global 
estimate of motion made in the course of the iteration preceding the iteration in progress, and 
from an additional displacement talsen from a pluraUty of predetermined additional 
S displacements; and 

- selecting the vector to be tested which yields the best match between the respective second 
block and the first block to obtain the local motion vector. 

It tiius that in the prior art, a plurality of iterations is carried out 

between the same images to obtain the final motion vci:tors corresponding to the highest 
10 resolution. 



It is, imer aUa, an object of the invention to provide a less complex motion 
vector estimation. To this end, a first aspect of the invention provides a method of estimating 
15 motion vectors as defined in claim 1. A second aspect of the invention provides a motion 
vector estimation device as defined in claim 10. A third aspect of the invention provides a 
video display apparatus as defined in claim 11. Advantageous embodiments are defined in the 
dqiendent claims. 

In a method of estimating motion vectors in accordance with a primary aspect 
20 of the present inventioni motion parameters are determined for a given field (n-l) of a video 
signal; and motion vectors for a field (n) of said video signal are determined in dependence 
upon at least one predetermined motion vector and at least one additional motion vector 

derived from S3dd motion parameters. 

These and other aspects of the invention wUl be apparent from and elucidated 
25 with reference to the embodiments described hereinafter. 



In the drawings: 

Fig. 1 shows linearly changing motion velocities due to zooming; 
30 Fig. 2 shows nine vectors taken from the temporal prediction memory of a 

motion vector estimator; 

Fig. 3 shows the mean squared errors (MSE) for the 3D-RS block matching 
motion vector estimation with and witiiout the new parametric vector candidate; 

Fig. 4 shows vector x-components without parametric candidate (upper half of 
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Fig. 4), and with parametric candidate (lower half of Fig. 4); 

Fig, 5 shows vector y-components without parametric candidate (upper half of 
Fig. 5), and with parametric candidate (lower half of Fig. 5); 

Fig. 6 shows an embodiment of a di^lay apparatus comprising a modon vector 
5 estimation device in accordance with the present invention; and 

Fig. 7 illustrates in a schematic form various embodiments of an image display 
apparatus having a motion vector furnishing device in accordance with the present inventicm. 



10 Motion estinutors minimize or maximize a, sometimes complex, but often 

simple, criterion function, selecting one of various possible motion vectors for every location 
in the image. They eitiier apply the exhaustive search metiiod, i.e. try all possible vectors in 
a predefined range, to obtain the global optimum of the criterion function, or use one of the 
efficient ^proaches and test a limited number of candidate vectors only [9,10,11,13]. In the 

15 more advanced efficient motion estimation methods this limited candidate set contains the 
most likely, prediction, vectors. The likelihood can be based on analysis of the picture at a 
higher hierarchical level [2,14], by analysis in the frequency domain [15,16,17], or the 
Ukelihccd is based on spatial and/or temporal proximity of the vectors in the candidate set 
[4,5,6,12,19]. 

20 Motion in video images is either due to object motion, or caused by camera 

movements. Object size causes correlation of motion vectors in tiie spatial domain, while 
object inertia explains temporal correlation of vector fields. The category of camera motion 
includes motion due to pans, tilts, and travels of tiie camera, and zooming with its lens. This 
type of motion usually causes very smooth velocities in the spatial and in the temporal 

25 domain. A zoom with the camera lens results in motion vectors that are linearly changing 
with the spatial position as illustrated in Fig. 1 in which Vv indicates linearly changinf 
vertical velocities and Vh indicates linearly changing horizontal velocities. A pan, tilt o? 
travel with the camera, on the other hand, causes a uniform motion vector for the entirt 
television screen. These types of motion can be described with a three parameter model, a.^ 

30 has been suggested before [3]. Also methods have been suggested in die literature how U 
extract the parameters of such a model from a rough motion vector field [1]. Even mon 
complex global motion, like rotation, can be described with a parametric model, which thai 
needs up to eight parameters [18]. In this description, we propose to use the rough estimat 
of the motion vector field in the previous image for generating, with a simple parametria 
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description of this field, an additional prediction vector for every location in the picture. This 
prediction is included in the candidate set of an advanced efficient block-matcher, that further 
includes temporal and spatial prediction vectors and an update Uiereof 14,6]. As tiie 
parametric model enables calculation of vectors even at the boundaries of the image and, if 
5 required, with sub-pixel accuracy, it can bring a considerable improvement in case of camera 
motion, larticularly if there are areas with Uttie or low-contrast detail near the boundaries of 
the image, where temporal and spatial predictors sometimes fail. 

Section n of this description introduces our notation and summarizes the motion 
10 estimation algorithm of [6], which we used as a basis to vaUdate our improvement. 
Section m introduces tiie parametric model and the extraction of the additional candidates 
from previously calculated motion vector fields. Section IV gives an evaluation of the 
improvement in a critical appUcation (de-interlacing), section V describes how Uie 
appUcability of MPEG motion vectors can be extended by using the principles of the present 
15 invention, and section VI summarizes our conclusions. 

n. The 3-D Re cursive Search Block-Matchfit 

As we considered perfection of an advanced motion estimator most interesting, 
we appUed tiie high quaUty metiiod of [6]. This algoritiim yields a quarter pixel accuracy and 
20 a close to true-motion vector field, relevant for scan rate conversion [4,5]. Furtiiermore, tiie 
efficiency of this algoritiim is such that it is currentiy tiie only single chip true-motion 
estimator [7,8]. 

In block-matching motion estimation algoritiims, a displacement vector is 
assigned to tiie center X of a block of pixels B(K) in tiie current field n by searching a similar 
25 block witiiin a search area SAQD, also centered at X but in tiie previous field n-1. The 
similar block has a center which is shifted witii respect to K over tiie displacement vector 
fifX. n). To find n), a number of candidate vectors Q are evaluated applying an error 
measure e(Q, X to quantify block similarity. 

More formally, CS=" is defined as tiie set of candidate vectors Q, describing all 
30 possible (usually integer) displacements witii respect to X witiiin Uie search area SA(K) in tiie 
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previous image: 

C5°^={£ I -WsC,s +W, -MiCyi +M) 

where N and M are constants limiting SAQQ. Furthermore, a block B(K) centered at K and of 
sucXbyY consisting of pixel positions i in the present field n, is now considered: 

The displacement vector Q,QL n) resulting from the block-matching process, is a 
candidate vector Q which yields the minimum value of an error function e(Q,}Ln)\ 

aa»)e{£€C5-"|fl(C Xn)io(Z.X. b)V£€C5"^} (3) 

If, which is the common case, the vector fife, n) with the smallest matching 
error is assigned to all pixel positions x in the block B(X): 

n)e{Qe CS^ | X, «) ^ a(£ X «) € CS^) 

rather than to the center pixel only, a large reduction of computations is achieved. As an 
implication, consecutive blocks BQO are not overlapping. 

The error value for a given candidate vector £ is a function of the luminance 
values of the pixels in the current block and those of the shifted block from a previous field 
summed over the block BQO- A common choice, which we will use, is the Sum of tht 
Absolute Differences (SAD): 

8(C B) = SAD = £ I J^i, n)-F(2-C «) I (5) 



Alternatively, a Mean Square Error (MSE) or a Normalized Cross Correlatio 
Function (NCCF) may be used. 

Rather than calculating all possible candidate vectors, the 3-D Recursive Searci 
block-matcher of [6], takes spatial and/or temporal "prediction vectors" from a 3-1 



SUBSTITUTE SHEET (RULE 26) 



wo 97/46022 pp-/IB97/00548 

6 

neighborhood, and a single updated prediction vector. This impUcidy assumes spatial and/or 
temporal consistency. The updating process involves updates added to either of the spatial 
predictions. We 5q)pUed a candidate set CSQL n) containing candidate vectors Q, from which 
the block-matcher selects its result vector^, defined by: 



(6) 



} 



5 where the two options for candidate vector £, alternate on block basis, and the update vector 
U(K. n) is taken from a Umited fixed integer update vector set US„ in our case: 



) 



To realize sub-pixel accuracy, the update set of equation (7) is extended with 
fractional update values. We reaUzed a quarter pixel resolution by adding the following 
fractional vectors to the update set: 



10 



Because of the smaU number of candidate vectors, tiic method is very efficient. 
Furthermore, due to the inherent smootiiness constraint, it yields very coherent vector fields 
that closely conespond to the true-motion of objects. This particularly, makes it suitable for 
scanning format conversion. 

15 nr. Camera m o t ion a n d the n ?Tamstric mffltel 

Camera motion includes motion due to panning, tilting, riding, and zooming of 
the camera. This type of motion has a very regular character causing very smooth velocities 
i.e. motion vectors, compared to object motion. Zooming with the camera will generate 
motion vectors that are linearly changing with the spatial position, as illustrated in Fig. 1. 
Panning, tilting or riding with a camera, on the other hand, will generate a uniform motion 
vector for the entire television screen. 



20 
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m.l A three narameter model 

These types of motion can be described with a three parameter model, as has 

been suggested before [3]: 



' Piin) ^ Piin)x\ (9) 

in which />, describes the panning, pi the tilting, and the zooming of the camera. This 
5 model is valid only if the blocks are exactly square on the image plane. At least in television 
this assumption is often invalid, even when the blocks are square on the pixel grid, due to 
the use of various aspect ratios with fixed sampling frequency. 

In these cases a four parameter model can be used according to: 

^^"^-[p^in)^p,(n)yj 

where a fixed ratio results between pj and p^, which is determined by the ratio of the 
10 horizontal (H) and vertical (V) ^pling density. If the ratio p^/pt does not correspond to the 

ratio of the H and V sampling density, the extracted parameters must be unreliable. It is 

possible therefore, by measuring this ratio, to identify situations in which the additional 

candidate is better not used in the motion estimator. 

Extending the model to a six parameter model renders it furthermore possible to 
15 include rotations. This type of motion, however, is not very likely in television broadcast 

material, and therefore not further considered in tiiis description (although not excluded from 

the claims). 

m.2. Up grading the 3-D T(S Biork-Mmch gr with a parametric candidatg 
20 An implementation of the proposal results by adding a single candidate vector 

for every block to the set CSPC, n) defined in section n, where this additional candidate 
£QL n) for the block BQO is calculated according to the four parameter model of equatior 
(10), i.e. it results: 



Rather than adding a candidate vector to Uie motion estimator it is possible tc 
25 alternate (e.g. on block base) this additional candidate with another candidate vector. In thi 
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case the operations count of the estimator hardly increases, while the advantage was found to 
be comparable. As a consequence candidate vector CU in equation (6) is modified to: 



mi + [27)' " " ^ ^ * ' 

A refinement of this thought is to have the occurrence frequency of the 
laiametiic model candidate depend on the reliabiUty of the model. In our implementation this 
5 led to a further modification of d according to: 

*P3(»)X) (oddblocksAmodelreliable) 
(else) 



10 



In the next subsection we will describe a parametric model extraction procedure 
that allows such a reliability indication. 

777.3. P,ytTartion of the parameters from thfi Mnags^ 

There are many options to extract the parameters of a global motion model 
from an estimated motion vector field. In our case where the model is integrated in the 3-D 
RS Block-Matcher, it makes sense to start from already available motion vectors, i.e. the 
vectors available in the temporal prediction memory. To keep the operations count low, it is 
furthermore attractive to use a Umited set of the vectors available in this memory only. 
15 As a consequence of the choice to use vectors from the prediction memory, the 

vaUdity of the candidates generated with the parametric model increases in time due to 
recursivity of this approach. If the parametric model is valid for significant portions of the 
image, the additional candidate will be selected by tiie estimator engine more often, which in 
tiie next field improves the quaUty of the parameters extracted from the vectors in the 
temporal prediction memory. This again improves the accuracy of the generated additional 
candidate, it wiU be selected more often, and tiie model will improve furtiier, etc.. 

To estimate the parameters of the model describing tiie global motion, we took 
a sample set S(n) containing 9 motion vectors, n-1) from different positions K on tiie 
block grid in a centered window of size (W-2m)X x {H-2q)Y in tiie picture with width WX 



20 



SUBSTITUTE SHEET (RULE 26^ 



wo 97/46022 PCT/IB97/00548 

9 

and height HY from the temporal vector prediction memory according to: 
S(n) = (aCX n - 1) 1 Jf, = <Wa-ni)X, 0. +(W72 " w)^ 

Xy=-(H/2-9)y, 0. +{HI2-q)Y) 



(14) 



where m and ^ are small and depend on the number of blocks in the image. Fig. 2 illustrates 
the position in the image from which the vectors in the sample set are taken. Choosing m and 
9 to be small, generally yields a more accurate estimate of the zoom parameters and p^. 
5 Vectors too close to the image boundary, however, can be unreliable in case of fast camera 
motion. It will be clear that other choices are possible that might prove to be better in 
specific cases, but we see no evident cause for systematical improvement with another set of 
similar size. Larger sample sets can improve the result, but also increase the calculational 
burden. 

10 A four parameter model can be solved with four equations, i.e. two 

"independent" sample vectors. Indq)endent, here, means that these vectors are not taken 
from the same horizontal or vertical position of the image. The number of combinadons tc 
choose r samples from a set of J is known from elementary statistics: 

lr\ s\ (15) 



,s} {s - r)lr\ 



, (r, J6N*) 



In our case we arrive at j=9 and r =2, which impUes 36-18=18 options to solve the model 
15 as there are 36 combinations of 2 out of 9, and 18 pairs of "dependent" samples, i.e 
samples on the same row or column. There are three rows and three columns, each enablin 
three pairs of two vectors being selected from the available 3 vectors, so 2x3x3=1 
dependent pairs. 

Extraction of the parameters from the i* pair of independent sample vectoi 
20 fi(X7. "-1) and B.(Ka. in S(n), is straightforward: 

PSCW) = y _y 



and: 



10 



15 
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while: 

I (18) 
piin) =DJiXu, Yu, B - 1) -P3(n)^u 

and finally: 

piiR) = D,(Xu, Tu, B - 1) - piCB) Y„ ^^^^ 

Now for every pair of two independent sample vectors a set of four parameters 
is avaUable. and the best parameter set has to be selected. We propose here to assign the 
median value of all options for each parameter to the eventual model parameter. 
Consequently, for i=l up to i=18 we rank the individual parameter values such that: 

Vipsiil? A V;ll ij i 4: Pj'\n) npjin) i P!'\n) 

and derive the final value of the model parameters according to: 
Vyi 1 . 4: p/«) = |(p;(B) - ip;\n) ) ^^^^ 

Finally, the ratio of p, and p, is calculated to check the reUability of the model. 
In the experiments we allowed 25% deviation from the nominal value. If the parameters pass 
this test and the ratio should be p^p, = R according to the H and V sampling densities (see 
section in.l), it is possible to sUghtiy further improve the accuracy of the model by 
replacing p, and p, by their average value after correcting for the different sampling densities 
( p, becomes: (p, + /?. p.)/2, and p, becomes: (p. + 1/R. P.)I2 )• This sophistication was 
implemented in the evaluated algorithm. 

In an alternative embodiment which is favorable as it requires less computations 
than the median algorithm of equations (20) and (21). the set of four parameters for every 
pair of two independent sample vectors arc used in the following manner. Hie 18 different 
sets of parameters are cyclically used (one pair for each field) until the model becomes 
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reUable, i.e until a correct p3/p4 ratio is obtained. The set which yielded the correct p3/p4 
ratio is maintained until it appears that the p3/p4 ratio is no longer correct. In that case, the 
parameter sets are again cyclically applied until the correct p3/p4 ratio is obtained again. 

5 There are many, options to extract the parameters of a global motion model 

from an estimated motion vector field. Furthermore, there are many possible motion 
estimation algorithms in which an additional candidate vector generated with such a 
parametric motion model could be applied. Another possible implementation of the invention 
in the 3-D RS block-matching motion estimator of [6] will be described bdow. 

10 To estimate the parameters of the global motion model, nine vectors, EI0..8] 

assigned to different positions in the picture are taken from the temporal vector prediction 
iriemory as illustrated in Fig. 2. Four vector values will be selected from this vector set to 
calculate the parameters. In order to select the most reliable base for the parametric model 
(the best four vectors out of the nine available ones), first three average x-components Djjq] 

IS are calculated according to: 

S^[l]=(l2,[ll+fipi)/2 
a^[3]=(fl^[5J+flJ6])/2 

To each of these averages a quality measure DiffJqJ is assigned using: 
Dijgrj.l]=absiQJl]-Etp]) 

DiffJ2] = absiQJl] - QJO] ) <23) 

The background of this quality measure is that the calculated differences shouU 
be zero if the vectors are only due to gl<*al motion (pan or zooming of the camera). Th> 
larger a certain difference is, the larger the chance that the corresponding vectors are a poo 
20 base for the parametric model. Similar calculations are required for the vector y-component: 

a^n]=(a,m+fly[8i)/2 

J2^^[2]=(fi'[61-^fi[41)/2 (24) 

fl'[3]=(a[31^fl[31)/2 

Ay y y 

and: 
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The vector x or y component values belonging to the smallest difference are 
used to interpolate/extrapolate the new parametric subpixel vector candidate using: 

X, = . ( m =1 V 3 V 5 . AV9: Diffjim] i Diffjiq] ) 

X ^DXm]Am^2y4y6,Mq:D{jfJim]iDiffJiq]) (26) 
y, - DJim] , ( m =3 V 4 V 8 , AV9: Diff^m] i Diffjiq] ) 

The gradient pj of the model must be calculated, as weU as the parameters Pi 
and P2 indicating the center of the zoom for respectively the vector x_component and the 

5 vector y.component. The gradient p„ is the same for the x_ and y_component, due to 
panning and zooming properties. The gradients are calculated separately. When the separate 
gradients are significantly different, the parametric vector candidates will be regarded 
unreUable and this information can be used to modify the occurrence frequency of the 
parametric candidate as mentioned above. To avoid small disturbances, both gradients are 

10 averaged, giving the final gradient pa. 



(yryu> \ (27) 

j 



where N, and are the horizontal and vertical distance measured on the block grid between 
the positions of the vectors on which the gradient calculation was based. The parameters p, 
and P2 can be found by extrapolation, making sure that the interpolation starts at the edges of 
the vector field: 

(28) 

y 

15 where X[5] and Y15] are the coordinates of D[5] on the block grid. The parametric vector 
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candidate (one per block) can then be calculated as: 

(29) 



which can than be added to the candidate vector set of any motion estimation algorithm. 



It is good to notice that the reliability criterion mentioned above has to be 
5 adapted in case of geometrical distortions in the processing between camera and motion 
estimator. These distortions (e.g. horizontal compression or vertical stretching) are 
sometimes introduced by circuits for wide-screen television. 

IV. Evaluation of the improvement 
10 The expected advantage of our proposal was an increased accuracy of the 

estimated motion vectors in case of camera motion. As the original 3-D RS block-matching 
algorithm is already very accurate, a critical application is required to show the 
improvement. We found in an earUer stage already [6], that de-interlacing requires a high 
accuracy from the motion estimator, i.e. the sub-pixel fraction of the estimate plays an 
15 important role in the de-interiacing quality. Therefore, here again, we appUed the 3-D RS 
block-matcher in the time-recursive de-interiacing algorithm of Wang et al. [20], to verify 
the advantage of the parametric candidate vector. We essentially calculate a motion 
compensated mean square prediction error, similar to what we used in [6], to get an 
indication of the quality of the motion vectors. 
20 More precisely, the MSE is calculated in the measurement window MW of 

(W-2m)X by 7 pixels, indicated in Fig. 2, between the luminance of the current input 

field Fli, n) and the previously calculated sequentially scanned picture F,(i. n-1) shifted over 
the estimated displacement vector QiZj n): 

where j runs through all positions in the measurement window on the pixel grid of the odd 
25 lines in an odd input field, and through all pixel positions of the even lines in case of an ever 
input field. 

In Fig. 3 we show the value of MSE(n) (vertical axis) for a number of field; 
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(horizontal axis) with (curve b) and without (curve a) the parametric candidate vectors in the 
estimator of [6], It can be clearly seen that both algorithms converge in the first fields, and 
that the parametric candidate gives a significant improvement of the performance in the later 
fields. The pictures were selected to provide critical test material. The tested sequence 
5 contains large areas with much vertical and horizontal detail, and movement in various 
directions with a large number of (sub-pixel) velocities due to zooming of the camera. It 
appeared that the additional candidate dees not degrade the performance in case no camera 
motion is present. We found that the MSE score on zooming or panning picture material, 
was some 12% lower with the new parametric candidate. 

10 

Figs. 4, 5 visualise the results of the proposal. In these figures the simulation 
results are shown, indicating the velocities with (lower half of Figs. 4, 5) and without (upper 
half of Figs. 4, 5) the parametric vector candidate. The vector x-component is shown in 
Fig. 4, whereas the vector y-component is shown in Fig. 5. The different shades of grey 1-6 
15 in Fig. 4 and 1-5 in Fig. 5 indicate mutually different motion vector component values. In 
this case the input for the model was a camera zoom which, as shown in Fig. 1, should 
result in linearly changing velocities in both the horizontal and the vertical directions. From a 
comparison between the upper and lower halves of Figs. 4, 5, it follows that due to the 
measure of the present invention, the motion estimator accuracy has been greatiy improved. 

20 

Fig, 6 shows an embodiment of a display apparatus comprising a motion vector 
estimation device in accordance with the present invention. A television signal received by an 
antenna A is applied to a processing circuit P for tuning, demodulation and video signal 
processing operations well known to those skilled in the art. An output of the processing 
25 circuit P is applied to a first field memory FMl for obtaining a video signal at a doubled 
field frequency. An output of the first field memory FMl is ^plied to a second field 
memory FM2 to obtain a field-delayed video signal. An input and an output of the second 
field memory FM2 are applied to a motion vector estimation arrangement ME to (*tain 
motion vectors B(x, n). The motion vector UilL. n) and the input and the output of the 
30 second field memory FM2 are applied to a motion compensated interpolator I, an output of 
which is zpphed to a display unit Dp. 

The motion vector estimator ME comprises an update generator UG including a 
mod p counter to which a block frequency clock signal Nj,, is applied. An output of the 
counter C is applied to a look-up table LUT to obtain update vectors iI(X, n) as described 
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above, and an update-dependent penalty PQD which increases with an increasing update 
vector length. The update vectors n) are appUed to an update and multiplex circuit UP- 
MUX which also receives a zero vector Q, a parametric candidate vector EYQO from a 
parametric candidate generator PG, and previously determined vectors Q(x+X, y-Y, n) and 

5 n(3t-X, y-Y, n) from a prediction memory PM. The update and multiplex circuit UP-MUX 
appUes three candidate vectors to a best matching vector selector SEL which also receives a 
vector y+2Y, n-1) from the prediction memory PM. The best matching vector selector 
SEL further receives the input and the output of the second field memory FM2, and 
determines the best vector B(X, n) between a block in a current field and a corresponding 

10 group of pixels in a previous field. The best vector UQL, n) is appUed to the prediction 
memory PM, and to a block erosion circuit BE to obtain a motion vector E(X. n) for each 
pixel X in the block X. From the prediction memory PM, the sample vectors shown in Fig. 2 
are appUed to a parameter calculator PC to obtain four motion parameters which are appUed 
to the parametric candidate generator PG. 

15 

V. Extending the applir^bilirv of WW motipn Yfifitprs. 

DE-A-195. 10.389 [21] discloses a method and a circuit arrangement for 
receiving coded video signals. The method concerns a video signal which contains a 
sequence of images with compressed redundancy, e.g. coded in accordance with the MPEG2 

20 standard. The motion decoding is carried out first. The post-processing of the motion 
decoded fields, which post-processing essentially contains interpolations, is carried out nex 
togetiier with tiie processing to remove fUcker. Advantageously, tiie motion vectors retrieve* 
at the motion decoding, are used also for tiie flicker removal. 

As mentioned above, motion estimation can be seen as an optimization problem 

25 a sometimes complex, but usually simple criterion function has to be minimized c 
maximized to find an output motion vector. There arc brute force methods that simply try a! 
possible vectors, in a predefined range, in order to be sure to obtain the global optimum c 
the criterion function. Also there are efficient approaches tiiat test only the most Ukei 
motion vectors. This UkeUhood is usually determined by spatial or temporal proximity, an 

30 consequentiy temporal and spatial prediction vectors have been popular in the efficic: 
motion estimation algorithms. Depending on the motion estimation algorithm used, U 
properties of the resulting motion vector fields are different, so that the appUcation areas f 
the various motion estimation algoritiims are different. Motion vectors can be used in hij 
quaUty video signal processing, like motion compensated scan conversion, motic 
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compensated noise reduction and image coding. The intended processing puts constraints on 
various quaUty aspects of the motion vectors. For example, high quaUty motion compensated 
scan conversion requires a motion vector field where the vectors represent the true motion 
vectors (i.e.. true direction and speed) within the image sequence. For coding appUcations 
5 this is irrelevant, as the only quaUty criterion is a low average prediction error. 

Brute force methods like full search block matching tend not to give true motion 
vectors. Instead, fuU search block matching minimizes the error between blocks, so its 
vectors will connect two blocks that have a minimal error between these two blocks, which is 
suitable for minimizing the prediction error in coding applications. WhUe these minimal 
10 errors work efficienUy for compression algoritiims like MPEG, the resulting vectors are 
unsuitable for use in high quality scan conversion. On tiie other hand, more efficient motion 
estimators like 3-D recursive block matching tend to give true motion vectors that can be 
used in high quaUty scan conversion but which may not be suitable for use in compression 
algorithms. 

15 With the emergence of MPEG and other types of video compression it is 

possible to have input video material that already contains motion vectors, so at first sight it 
may seem that for this type of source material motion estimation at the receiver side is not 
necessary. However, it is not certain that these motion vectors can be directiy used for e.g. 
scan rate conversion, because it is unknown whether die MPEG motion vectors represent 
20 true motion or not. At the decoder/receiver side it is unknown what type of motion 
estimation algorithm was used at the encoder/transmitter side, so one must assume, as a 
worst case situation, that the MPEG motion vectors are optimized for an efficient 
compression and that they do not represent true motion vectors. For example, periodical 
structures and noise in picture areas wiUi UtUe detail, may cause such inconsistent vectors. 
25 Another problem with motion vectors in MPEG data streams is that it is uncertain tiiat aU 
motion vectors are transmitted witiun the data stream. For these reasons, die technique 
proposed by [21] seems not to be suitable for high quality post-processing. 

In an attempt to solve a similar problem, it has been proposed [22] to smooth 
the vectors transmitted with a digitally encoded television signal before using these vectors 
30 for a field rate upconversion operation. 

It is an object of a further aspect of the invention to furnish motion vectors 
which are suitable for high quality post-processing. This further aspect of the present 
invention is based on the recognition that the problems described above do not exclude the 
MPEG motion vectors entirely from use in high quality scan conversion. When an 
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appropriate post-processing is appUed, the MPEG motion vectors can be made useftil. More 
specificaUy, when the receiver is able to determine the quality of the MPEG motion vectors, 
and when it is able to improve the quality of the MPEG motion vectors so that they meet 
certain criteria for the intended processing, the MPEG motion vectors can be used. Several 
5 possible post processing methods to extend the applicabUity of the MPEG motion vectors are 
considered: 

1. Process the MPEG motion vectors with an operation like some sort of vector 
smoothing operation to increase the usability of the MPEG motion vectors. 

2. Use the MPEG motion vectors as additional candidates along with spatial and/or 
10 temporal candidates in, for instance, a 3-D recursive block matching algorithm, similar to the 

use of an additional parametric vector candidate as disclosed in the non-prepublished EP 
priority patent appUcadon no. 96201462.7 filed on 24.05.96 (attorneys' docket PHN 15,824). 

3. The MPEG motion vectors can be used to calculate the parametric vector 
candidates described in the non-prepublished EP patent appUcation no. 96201462.7. 

15 For a schematic description of the various embodiments, reference is made to 

Fig. 7. 

Fig. 7 shows a basic section B, and four sections El .. E4 showing four distinct 
ways to augment the basic section B to obtain four distinct embodiments B+El, B-i-E2, 
20 B+E3, and B+E3+E4 of an image display ^paratus. 

In the basic section B, an MPEG signal comprising a video signal V and motion 
vectors MV, is appUed to a decoder DEC in order to retrieve the video signal V in 
dqiendence upon the motion vectors MV from the MPEG input signal. The video signal V is 
^lied to a video post-processor VPP to obtain an ou^ut video signal OVS having i 
25 doubled field frequency and/or line frequency or an improved motion portrayal with respeci 
to the video signal V. Altemadvely or in addition, the video post-processor VPP reduce; 
noise and/or changes the number of pixels per line and/or lines per field of the video signa 
V. The video post-processing carried out by the processor VPP is dependent on improve, 
motion vectors MV'. The output video signal OVS is displayed on a display device D. 
30 In accordance with a first embodiment B-J-El, the improved motion vector. 

MV' are obtained from the motion vectors MV contained in die MPEG signal by means of 
post-processor PP. For example, a smoothing operation wiU reduce most of the artifact 
which would arise if the MPEG motion vectors MV were direcdy used for die video posi 
processing carried out by the processor VPP. A smooUiing operation may include separat 
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low-pass filtering operations on the horizontal and vertical components of the motion vectors. 
Alternatively, the horizontal and vertical components of the motion vectors can be applied to 
median filters to replace extreme motion vector component values by existing motion vector 
component values of motion vectors of neighboring blocks of picture elements. 

5 In accordance with a second embodiment B+E2 which is covered by a 

copending patent appUcadon (attorneys' docket PHN 16,106), the MPEG motion vectors MV 
are multiplexed by a multiplexer MUX with candidate vectors generated by a candidate 
vector generator CVG, optionaUy after a (e.g. smoothing) post-processing operation carried 
out by the post-piccessor PP. The multiplexed vectors are appUed to a candidate vector 

10 selector CVS to obtain improved motion vectors MV' which are expected to show the 
smallest errors. 

In a third embodiment B-J-E3 which is covered by the claims of the present 
patent appUcation, a jarameter extractor PE extracts motion parameters pi .. pn from the 
MPEG motion vectors MV. The motion parameters pi .. po are appUed to a vector field 
15 calculator VFC to obtain the improved motion vectors MV'. The predetermined motion 
vector recited in claim 1 may be the zero vector. 

In a fourth embodiment B-t-E3-i-E4 which is covered by the claims of the 
present latent appUcation, tiie motion vectors calculated by die vector field calculator in 
section E3 are multiplexed with candidate motion vectors from die candidate vector generator 
20 CVG, and appUed to the candidate vector selector CVS to obtain the improved motion 
vectors MV'. This fourth embodiment combines the feahires of the second and third 
embodiments. 

The basic considerations underlying U\e present aspect of the invention can be 
summarized as foUows. Different motion estimation algorithms vdll have different 

25 appUcations based on the properties of tiieir resulting vector fields. MPEG video data streams 
contain motion vectors, which can possibly be used in high quaUty video pnjcessing. 
However, it is Ukely that tiiese MPEG motion vectors are optimized to achieve an efficient 
compression, so that diese vectors cannot be expected to represent the trae motion. This 
means that additional processing is needed to improve their quaUty and usabiUty for a high 

30 quaUty post-processing. It is proposed to process tiie MPEG motion vectors at the receiver 
side. More specifically, the MPEG vectors are used to generate motion parameters which are 
used to obtain one or more parametric motion vectors which can be used as additional 
candidate vectors in an efficient motion estimator. In this manner, the appUcabUity of MPEG 
motion vectors is extended. Obviously, the present aspect of the invention is not Umited to 
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MPEG signals; other signals can be treated in the same manner. Reference is made t 
copending patent applications for features disclosed but not claimed in this application. 

5 A new motion estimator has been presented. This estimator selects its motic 

vector for the current image from four candidate vectors only, and is characterized in th- 
one of these candidate vectors is produced with a parametric model describing the glob; 
(camera) motion in the previous image. 

Wc introduced this "parametric candidate" in a very efficient (3-D Recursiv 

10 Search) block-matching algorithm. To this end, we block-altematingly replaced one of ti: 
candidates of this estimator by a candidate generated with a four parameter model. Th 
parameters for the model were extracted once per field using nine widely spaced ^mipl 
vectors only, taken from the previously estimated vector field. This results in an almo 
negligible additional processing power requirement for the eventual algorithm. We sbowe 

15 that, with these 9 extracted motion vectors, it is possible to generate 18 sets of 4 paramete: 
describing the camera motion. A median operation was introduced to select the applie 
parameter set, thus eliminating the outliers due to object motion. Furthermore, we showe 
that knowledge of the horizontal and vertical sampling densities could be used to judge tl 
reliability of the model. This infonnation provided a means to switch-off the bloc; 

20 alternating injection of the "parametric candidate** in the estimator in case it could lead to 
degradation of the estimator performance. 

In the evaluation part IV, a significant advantage, up to 50% reduction in MS; 
was found on critical material applying the motion vectors for de-interlacing. A photogra] 
of the vector field enabled a subjective evaluation, suggesting that also the fractional part 

25 the displacement vectors obtained from the resulting block-matcher closely correspond to t 
true-motion in the sequence. Finally, it was shown that the performance of the reliabiL 
indicator, although working properly in the experiments, was not too critical. 

A preferred embodiment of the invention, providing an efficient true-mod- 
estimator using candidate vectors from a parametric motion model, can be summarized 

30 follows. Efficient motion estimation algorithms are known that select their output moti 
vector from a limited number of likely correct candidate, or prediction, vectors. In additi 
to the known spatial and temporal prediction vectors, an additional and independe 
prediction is proposed. This candidate is generated with a parametric model describing t 
global motion in a previously estimated motion field. The proposal is elaborated as 
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addition to the 3-D Recursive Search block-matching algorithm. The evaluation shows that a 
sub-pixel accurate, true-motion estimator results with a very low operaticwis count. 

It should be noted that the above-mentioned embodiments illustrate rather than 
limit the invention, and that those skiUed in the art will be able to design many alternative 
5 embodiments without dq>arting from the scope of the appended claims. In the claims, any 
reference signs placed between jmentheses shall not be construed as limiting the claim. The 
invention can be implemented by means of hardware comprising several distinct elements, 
and by means of a suitably programmed computer. 

In the present disclosure, it has been described to use parameters determined on 
10 the basis of motion vectors obtained for a preceding field n-1, to obtain motion jmameters 
used in determining an additional vector for the present field n. However, in a conversion 
from a 50 Hz input signal originating from a 24 Hz movie film, to a 100 Hz output signal, 
motion vectors determined during a first estimation stage for a given field may be used to 
obtain parameters for use in a second estimation stage for the same field. Thus, it may 
15 h^pen that field n-1 is the same field as field n. 

While a preferred jq)plication of the present invention is in a recursive motion 
estimator, the additional motion parameter-dependent vector may also be applied in other 
motion estimators like the well-known 3-step block matching algorithm. Thus, the expression 
predetermined motion vector in the claims includes candidate motion vectors obtained in 
20 accordance with any algorithm other than an algorithm in which motion vectors arc derived 
from motion parameters. 

In another embodiment in accordance with the present invention, the motion 
parameters are obtained in accordance with a 2-D histogram algorithm to which all motion 
vectors for the given field (n-1) are applied, the peaks in the histogram forming the motion 
25 parameters. The peaks indicate motion vectors which very often occur in the image and 
which probably describe a global motion. The motion vectors indicated by the peaks, and/or 
points of gravity of a cluster of peaks, can be used as additional parametric moticm vectors. 
If there are more peaks than additional motion vectors required, the motion vectors indicated 
by the peaks can be used in a cyclic manner as additional motion vectors. 
30 The motion vector estimation in accordance with the present invention is 

preferably applied in a scan format conversion for e.g. converting a 50 Hz field frequency 
signal into a 100 Hz field frequency signal, for converting a 60 Hz field frequency signal 
into a 50 Hz field frequency signal, or for converting an interlaced signal into a 
progressively scanned signal. 
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Claims: 

1. A method of estimating motion vectors, comprising the steps of: 
determining (PC) motion jmametcrs (pl-p4) for a given field (n-1) of a video 

signal; and 

determining motion vectors ( QQbi n) ) ^or a field (n) of said video signal based 
5 upon at least one predetermined motion vector ( D(x-X, y-Y, n), Q(x+X, y-Y, n), 
E(x, y+2Y, n-1) ), and at least one additional motion vector ( VYCX) ) derived (PG) from 
said motion parameters (pl-p4). 

2. A method as claimed in claim 1, wherein said at least one predetermined 
10 motion vector ( Q(x-X, y-Y, n), Q(x+X, y-Y, n), y+2Y, n-1) ) includes at least a 

first previously determined motion vector ( B(x, y+2Y, n-1) ) corresponding to a selected 
block of pixels out of a plurality of spatio-temporally neighboring blocks, and a second 
motion vector obtained by adding (UP-MUX) an update vector ( UQL, n) ) to the first or 
another previously determined motion vector ( Q(x-X, y-Y, n), B(x+X, y-Y. n) ). 

15 

3. A method as claimed in claim 1, wherein said motion vectors for a subsequent 
field (n) of said video signal are based (SEL) upon a selection from said at least one 
predetermined motion vector ( Q(x-X, y-Y, n), Q(x+X, y-Y, n), B(x, y+2Y, n-1) ), and 
said at least one additional motion vector ( PYfX) ) derived (PG) from said motion 

20 parameters (pl-p4). 

4. A method as claimed in claim 1, wherein said motion parameters (pl-p4) fo: 
said given field (n-1) of a video signal are determined (PC) from motion vectors determinet 
for said given field (n-1). 

25 

5. A method as claimed in claim 4, wherein said motion parameters (pl-p4) arv 
calculated (PC) by determining for each motion parameter, a plurality of paramete 
candidates from selected motion vectors, and by taking a median of said plurality c 
parameter candidates to obtain the corresponding motion parameter. 

30 
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6. A method as claimed in claim 4, wherein said motion parameters (pl-p4) are 

calculated (PC) by the steps of: 

calculating a plurality of horizontal and vertical difference vector components 
from a plurality of sdected motion vectors in the given field; 
5 selecting horizontal and vertical vector components corresponding to minimum 

difference vector components to obtain selected horizontal and vertical vector components; 
and 

calculating said motion parameters in dependence upon said selected horizontal 
and vertical vector components. 

10 

7. A method as claimed in claim 4, wherein said motion parameters (pl-p4) are 
calculated (PC) by determining for each motion parameter, a plurality of parameter 
candidates from selected motion vectors, and by cycUcally applying said plurality of 
parameter candidates to obtain the corresponding motion parameter until the parameters 

15 correspond to an aspect ratio of said video signal. 

8. A method as claimed in claim 4, wherein said motion parameters result from a 
two-dimensional histogram operation on said motion vectors determined for said given field 
(n-1). 

20 

9. A method as claimed in claim 1, wherein said motion parameters (pi .. pn) arc 
determined from motion vectors (MV) received with said video signal (MPEG). 



10. A motion vector estimation device (ME), comprising: 

25 means (PC) for determining motion parameters (pl-p4) for a given field (n-1) of 

a video signal; and 

means (SEL) for determining motion vectors ( n) ) for a field (n) of said 
video signal based upon at least one predetermined motion vector ( D(x-X, y-Y, n), 
]2(x+X, y-Y, n), U{x, y+2Y, n-1) ), and at least one additional motion ( EVQD ) "^^^^ 
30 derived (PG) from said motion parameters (pl-p4). 
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11. A video display apparatus, comprising: 

an input (A) for receiving a video signal at a first scan format; 

a motion-compensated interpolation device (ME, FM2, I) coupled to said input 
for providing an ou^ut video signal at a second scan format different from said first scan 
format; and 

a display (Dp) for displaying said output video signal at said second scan 

format; characterized in that 

said motion-compensated interpolation device (ME, FM2, I) includes a motion 

vector estimation device (ME) as claimed in claim 10. 
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